Scheduling Research Grant Proposal Evaluation Meetings

In many funding agencies a model is adopted whereby a fixed panel of evaluators evaluate the set of applications. This is then followed by a general meeting where each proposal is discussed by those evaluators assigned to it with a view to agreeing on a consensus score for that proposal. It is not uncommon for some experts to be unavailable for the entire duration of the meeting; constraints of this nature, and others complicate the search for a solution. We report on a system developed to ensure the smooth running of such meetings. 1 Background The process of evaluating research grant proposals presents some interesting opportunities for operations research practitioners and researchers. The model we discuss here assumes that a fixed pool of evaluators exists and the set of grant proposals is distributed amongst them subject to the evaluators’ stated abilities to evaluate each. (Although this step of the process is outside the current scope it is an interesting assignment problem with many side constraints. For example, some evaluators, such as vice-chairs, may be expected to take a reduced load of evaluations due to other duties; no proposal should have a majority of vicechairs evaluating it; and, for each proposal, one evaluator should be appointed as proposal reporter amongst the – usually 3, although requests for larger financial sums necessitate more – evaluators assigned to it with this extra duty evenly allocated amongst all evaluators.) In the present context our interest begins when a general panel meeting brings all of the evaluators together. Each proposal is discussed face to face by the assigned evaluators for the purpose of agreeing a consensus position and category scores, from which the final evaluation report may be written; the consensus meeting runs for fixed length of time. Following the entry of all scores in a database a ranking list is generated which forms the basis for funding decisions by the grant agency. So that the entire panel of evaluators may agree to the ranking list it is desirable that all consensus meetings be completed as quickly as possible allowing time for the inevitable clean-up before the final ranking list acceptance process. In its most restricted form the problem may be expressed as, given an assignment matrix of proposals to evaluators, where each proposal has been read by some subset of evaluators, generate a schedule of consensus meetings that uses 1 A panel may be thought of as a general research area, e.g. computer science. E. K. Burke, H. Rudová (Eds.): PATAT 2006, pp. 432–436. ISBN 80-210-3726-1. the fewest number of time periods where the meetings can be held. A maximum of T time periods exist. The constraints that must be respected, then, are: 1. A consensus meeting can only take place during one time slot; 2. An expert can only be in one consensus meeting during a time slot; 3. If a consensus meeting for a proposal takes place then all of the experts assigned to it must be present; 4. No more than T time slots may be used The goal is to minimize the number of time slots used. While the problem bears some resemblance to the teacher-class timetabling problem, the objective function differs, as well as some of the constraints. In the following section we describe our first approach to solve the problem by modelling it as an ILP. 2 An ILP Model Given E = eij , the assignment matrix that indicates what experts have been assigned to read proposal j, we can view its transpose E = P = pij as the matrix that indicates what proposals have been assigned to expert j. We introduce binary variables xij that indicate that the consensus meeting for proposal i will take place in time slot j, and yij that indicate that expert i is in some meeting during time slot j, 1 ≤ j ≤ T . Constraint 1 above can be implemented by